Testing software systems belonging to organizations is often a time consuming and expensive process. This is especially true when involving complex software systems such as Enterprise Resource Planning (ERP) systems, which have become the computational backbone of many business organizations. Such software systems often include a large number of software modules that need to be frequently validated to ensure that the software systems are operating as expected. Implementing configuration changes to software systems are often considered be risky and require regression testing to validate affected configuration elements (e.g., business processes impacted by the changes). The cost and time of a full regression test may be too high.
Deciding which tests to run (and what those tests should involve) in order to validate a configuration change is not a trivial task. Often, configuration changes may have unintended and/or unexpected side effects, which a tester may not be aware of. Since tests are usually designed and run by each organization separately on the organization's systems, this means that each organization must build up the necessary knowledge, and independently discover important aspects that should be tested when validating different configuration changes. Gaining this knowledge may require much effort and experience; in the meantime, testing the software systems may be a less effective and prolonged process. However, organizations that are able to utilize each other's testing-related knowledge, which is in a sense a wisdom of the crowd (of testers), may be able to come up with a more effective and efficient testing plan.
Collecting testing data from multiple organizations may not be that helpful if there is no way to sift through the massive amounts of data that can be collected and select appropriate tests for the task at hand. Additionally, not all tests are equal. While some organizations may have knowledge of how to test certain aspects of a software module (e.g., effective tests for testing a certain configuration change), others may have devised erroneous, ineffective, and/or irrelevant tests.